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ABSTRACT 

This paper describes the use of a theory-based 
framework, the Concerns-Based Adoption Model (CBAM) , in evaluating 
the implementation of three educational innovations in the Portland 
(Oregon) Public Schools during 1986-87: (1) Project READ, a 
districtwide alternative language arts program for grades 1-8; (2) 
Timeline, project management software for Evaluation Department 
planning, and (3) a K-8 social studies inservice program in a cluster 
of 3chools. It investigates the uses of three diagnostic dimensions 
of CBAMj Innovation Configurations (IC), Stages of Concern (SoC) , and 
Levels of Use (LoU) , and discusses their function in planning, 
monitoring, and evaluating these educational programs. The experience 
of using the Concerns Based Adoption Model in formative evaluation of 
these programs indicates that it can assist the evaluator in defining 
program elements and interpreting related teacher concerns and use, 
and that the data can guide the program staff in monitoring clients' 
use of the innovation, designing practical intervention strategies, 
and confidently attributing outcomes to the program. A bibliography 
xs included, and instrumentation is appended. (TE) 
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Introduction 

Educational program evaluation is a complex phenomenon that 
is often little understood by the participants it aims to help. 
While the evaluation process aims to improve student achievement 
and increase program effectiveness, too often these efforts are 
hampered by the realization that curriculum programs have not 
been faithfully implemented or have been so modified that they 
hardly resemble the original program. 

This paper describes the use of a theory-based framework, 
the Concerns-Based Adoption Model (CBAM)l' in evaluating the 
implementation of educational innovations. The paper reports 
applications of CBAM in curriculum and administrative programs. 
It investigates the uses of three diagnostic dimensions of CBAM: 
Innovation Configurations (IC) , Stages of Concern (SoC) and 
Levels of Use (LoU) and discusses their function in planning, 
monitoring, and evaluating educational programs. 



The Concerns- Based Adoption Model was developed at the 
Research and Development Center for Teacher Education, University 
of Texas-Austin. For more information, see Loucks, Susan & Hall, 
Gene E. (1977). Assessing and Facilitating the Implementation of 
Innovations: A New Approach. Educational Technology . 2J., 18-21. 
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Background 



Early on program evaluators learn that before they evaluate 
an educational program, they must determine whether or not the 
program has been implemented. Recognizing this need, the 
Portland Public Schools used the Concerns-Based Adoption Model to 
define program fidelity and describe the level of implementation 
of programs during 1986-87. The CBAM provides a practical and 
objective measure for assessing instructional change. Evaluation 
Specialists were trained in stages of Concern and Levels of Use 
methodology. CBAM helped us study the affective and behavioral 
changes at the district, cluster, and department level. CBAM 
diagnostic measures were used to assess the implementation level 
of three diverse programs: 

1) Project READ, a districtwide alternative 
language arts program for grades 1-8, 

2) Timeline, project management software for 
Evaluation Department planning, and 

3) A K-8 social studies inservice program 
in a cluster of schools. 

In each case, the author reports how the CBAM components 
helped describe the extent to which programs had been 
consistently implemented and what the related concerns were in 
the implementation process, with these data, the evaluator would 
know whether student outcome gains were attributable to the 
program, to some elements of the program, or to other external 
factors and also what kinds of concerns program staff had. 



The experience of using the Concerns-Based Adoption Model in 
program evaluation indicates that it can assist the evaluator and 
the program staff. It helps an evaluator define program elements 
and interpret related teacher concerns and use. The data guide 
the program staff in: 1) monitoring client use of the innovation, 
2) designing practical intervention strategies, and 3) being 
confident that outcomes can be attributed to the program. 

Pro-iect RFAn 

The first innovation to be presented is Project READ, an 
alternative language arts program for low achieving students in 
grades 1-8. This curriculum approach uses direct instruction 
with a focus on phonics, comprehension, and written expression in 
multi-sensory learning. The curriculum provides sequenced skill 
development and is integrated with the basal reading program. 

During the 1986-87 school year, 46 teachers in grades one 
through eight at 14 schools were involved in Project READ. These 
teachers utilized Project READ within their reading and language 
arts programs. In addition, four Project READ teacher-trainers 
conducted weekly demonstration lessons in each classroom 
throughout the year. Teachers implemented Project READ in order 
to use an alternative teaching strategy with low performing 
reading and language students, to improve student achievement, 
and to reduce student referrals to special education progiams. 



Evaluating the Tmplementat- ion of Proiec-t- r^ ah 

To assess the teachers' attitudes and use of Project READ, three 
components of the Concerns-Based Adoption Model were used: 
1) an Innovation Configurations checklist provided the basis for 
classroom observations, 2) stages of Concern questionnaires were 
administered to all teachers, and 3) Levels of Use interviews 
were conducted with a representative sample of teachers. 

Innovation Conf iguyat \ on ^ - The evaluator conducted classroom 
observations to record descriptions of teacher behavors and 
lesson content. This narrative was the basis for developing a 
preliiainary checklist to describe the key elements of the READ 
program. These key factors were shared with the developer and 
demonstration teachers to clarify the critical elements of the 
innovation and to develop a prototype configuration checklist. 
The checklist, displayed in Figure A of the appendix, will be 
used in subsequent monitoring of Project READ. 

^- Staaeg — of — Concern - The stages of Concerns (SoC) 
questionnaire (see appendix) is a 35-item instrument. It is 
designed to identify concerns typically associated with change. 



T Susan, Stiegelbauer, Suzanne M. , Hall, Gene E., and 

SSnSn,', Jo ^^"^ i^^lV* Measuring Innovation ronf jguration^. 
SggfgT^ J"*^ J^^Pl^y^^l^.^" - Austin: Research and Development 
Center for Teacher Education, University of Texas. 
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Table 1 disr ays the seven developmental stages of Concern^ with 
related definitions. In this context, "concern" does not imply 
negativity toward the innovation, but refers to a normal 
developmental pattern which occurs in the change process. 



Table 1 

Stages of Concern About the Innovation 



0 Af^AREtJSSS t Little concern about or involvement with the innovation is indi- 
cated. 

1 irtTOFfiATIOUAL i h general awareness of the innovation and interest in learn- 
ing acre detail about it is indicated* The person seems to be unworried 
about herself /himself in relation to the innov«iwj.on« She/he is interested 

in substantive aspects of the innovation in a selfless manner such as general 
characteristics » effects, and requirements for use* 

2 PZPSOttAl i Ind /idual is uncertain about the demands of the innovation, her/ 
his inadequacy to meet those demands, and her/his rnle with the innovation* 
This includes juialysis of her/his role in relation to the reward structure 
of the organization, decision making, and consideration of potential con- 
flicts with existing structures or personal commitment* Financial or status 
implications of the program for self and colleagues may also be reflected* 



3 MAUAGZ^ttrt ; Attention is focused on the processes and tasks of using the 
innovation and the best use of inforaati-^n and resources* Issues related to 
efficiency, orga/.ising, managing, scheduling, and time demands are utaost* 



4 CONSZQUZSCI : Attention focuses on intact of the innovation on students in 
her/his icanediate sphere of influence* The focus is on relevance of the in- 
novation for studentr, evaluation of student outcomes, including pt«rfor?\ance 
and competencies, anw changes needed to increase student outcomes* 



S COlLASORATIOlf : The focus is on coordination and cooperation with others re- 
garding use of the innovation* 



6 RSrocvsiifG ; The focus is on exploration of more universal benefits from the 
innovation, including the possibility of major changes or replacement with a 
more powerful alternative* Individual has definite ideas about alternatives 
to the proposed or existing form of the innovation. 



3 Hall, G.E., Wallace, R. C, Jr., & Dossett, W. A. (1973). 
A ggy^lPPingnwl conceptualization of i^Ha ad o ption p ir ocess within 

gqugatjgnal institutions, Austin: Research and Development 

Center for Teacher Education, The University of Texas. 
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The SoC questionnaire was senr to all Project READ teachers 
in December 1986 and again in December 1987. The purpose of the 
data collection was to profile the user group in terms of affect 
or developmental concern about using Project READ. Forty-six of 
the first year questionnaires were returned; 22 of the second 
year questionnaires were returned. 



Figure l presents the group profile of Project READ 
teachers' concerns at each stage for year one and year two. The 
percentile score represents the magnitude of concerns at each 
stage; the higher the percentile the more intense the concern. 

As individuals learn about and begin using an innovation, 
the model assumes that their concerns move in a developmental 
pattern from low to higher stages. m other words, the CBAM 
finds that individuals shift from high Stages 0, 1, and 2 scores, 
to high stage 3 scores, and finally, to the higher stages. After 
using Project READ for three months, the teachers' Stages of 
Concerns were high at Stage 0-Awareness, Stage 2-Personal, and 
Stage 5-Collaboration. It is characteristic of new users to have 
high awareness and personal concerns about an innovation. 
Because READ requires sustained training and coordination between 
classroom and demonstration teachers, it is reasonable to see 
high Collaboration concerns. The SoC profile identifies nonusers 
who are just becoming aware of READ and who have strong personal 
concerns about the project and its consequences for them. 
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Figure 1 



TEACHER CONCERNS PROFILE 
ABOUT PROJECT READ OVER TWO YEARS 





similar results are shown for year two, but at a less intense 
level. This suggests that while Project READ teachers have a 
personal interest in the innovation and its consequences, they 
have remained open and interested in the innovation. To reduce 
Stage 2 Personal concerns, the Project READ staff should plan 
staff development activities to help the teachers look at the 
innovation more objectively. 

c. L§ygls__of_Ltefi^ - The evaluators' goal was to assess the 
implementation level of Project READ at the end of the first 
year. To this end. Levels of Use interviews were conducted by 
certified LoU interviewers with a representative sample of 18 
teachers. Tne LoU interview is presented in the appendix. The 
LoU interview is a focused protocol that elicits information 
about specific behaviors associated with using an innovation, 
rhe tape-recorded interviews lasted about twenty minutes; then 
the tapes were rated by two interviewer/raters who independently 
assigned the level of use. Overall LoU was determined by 
agreement of two raters. Of the 18 tapes, 13 were agreed on by 
the first two raters (72%); four tapes required a third rating 
(22%); and one tape was rated four times. The inter-rater 
reliability ranged from 60 to 80% on the overall Level of Use. 



Loucks, Susan P., Newlove, Beulah W. , and Hall, Gene E. 
• Measuring Levels of Use of the T nnovatiQn. a Manual fn-r 
Trainers. Tnterviewi^rs. and Rafpr^, 2i„or^;T\^T^j:;L'^^^r.."!!g'^;^3^j^"^ 
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Table 2 presents the distribution of Levels of Use for the 
Project READ sample of teachers. 



Table 2 

Distribution of LoU for Representative Sample of 
Project READ Teachers 



Level 
0 


I 


II 


III 


IVA 


IVB 


V 


VI 


N 
% 






8 

(47%) 


7 

(41%) 


2 

(12%) 







As indivi luals move from nonuse of an innovation to first 
use, it is hypothesized that their level of use develops 
sequentially through the levels. Research indicates that the 
majority of first year users are at Level Ill-Mechanical Use. 
Nine of the READ teachers (53%) were rated above Level III; 47% 
of the first year READ users (N=8) wer-» rated at Level III. One 
reason that a larger percentage of first year users were rated at 
higher levels of use is because Project READ is an innovation 
which requires ongoing collaboration between the user and the 
demonstration teacher. Because READ is not a supplementary 
program, it may allow more rapid integration of the innovation 
with other elements of a teacher's instructional repertoire. 

- 9 - 
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In assessing the first year implementation*, the Levels of 
Use interviews provided data on the need for program support. 
The interviews also validated the relationship between Project 
READ theory and practice, so the evaluator could be fairly 
confident she was measuring the results of program instruction. 
LoU interview transcripts were reviewed to identify teacher 
response patterns at different levels of use. This process 
provided operational definitions of the use of Project READ. 

Table 3 displays the teacher-reported behaviors in two 
categories, management/logistics and instruction. The behaviors 
are organized by Level of Use. Generally, teachers at Level lll- 
Mechanical Use answered the questions by describing the details 
of their implementation, e.g., how the program was scheduled, 
group size, and management or materials issues. These teachers 
asked questions to which they were still seeking answers, e.g.. 
What is the appropriate pace of program delivery? How can I best 
integrate READ with other programs? The typical LoU III teacher 
commented on student bt »iavior in the classroom and speculated on 
ways to improve classroom management, e.g., instituting token 
reinforcement techniques. These teachers confined their use of 
Project READ to the target low achieving students. 

In contrast, teachers at Level IVA-Routine Use and Level IVB 
Refinement responded to the interview questions with specific 
references to student learning in Project READ. They reported 

- 10 - 
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T-ble 3 

.Project READ Levels of Use and Teacher-Reported Behavioral Definitions 



Levf*^ of Use 



III Mechanical 

User focuses on short- 
term, day-to-day use; 
changes made more to meet 
user needs than client 
needs ;user typically 
implementing innovation 
step-by- step. 



Teacher -Reported Behaviors 
Management/Logi st ic 3 Tns t r uct i on 



b. 



c. 



d. 



a. Used Project Read just 
with target students. 
Expressed need for 
additional materials; 
handouts, student read- 
ing booI;s. 

Spent a lot of time, 
effort on planning, 
materials organization. 
Instituted token manage- 
ment system in class - 
rom. 



a. Attempted to follow 
modeled instruction 
closely; did not vary 
instructional pace even 
when some students 
seaned to need more 
time. 



IVA .Routine 

Use of innovation stabil- 
ized; few changes made; 
little preparation or 
thought given to 
improving use. 



a. Added some of own in- 
structional strategies. 

b. Integrated instruction 
with other curriculum 
programs, eg. social 
studies, writing, 
literature; selected 
supportive reading 
materials with curric- 
ulum integration in 
mind. 



IVB Refinement 



User varies innovation 
somewhat to increase 
student outcomes; user 
considers both short/ 
long-term consequences for 
students. 



a. Participated in school 
planning to maintain 
students in Project 
READ during successive 
years. 



a. Regrouped based on 
student performance; 
used with large groups; 
planned cross-graded 
instruction; sometimes 
used with whole class 
including nontargeted 
students. 

Increased instructional 
time by enlisting 
volunteers for rein- 
forcement lessons. 
Increased review during 
instruction; provided 
extensive literature 
collection for regular 
take-home; extended 
process/skills use 
beyond language arts 
to real-life situations. 



b. 



c. 



MC 
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making minor modifications in the program to meet perceived 
student needs. These teachcirs talked about Project READ from a 
personal instructional perspective rather than in terms of 
program details. Teachers at LoU IVA and IVB used Project READ 
strategies with all of their students, not just target students. 

Results of a comparison of Project READ individuals' SoC and 
LoU found no significant relationship between the teachers' 
assessed concern and level of use. Teachers rated at LoU IVA- 
Routine and LoU IVB-Ref inement had high percentile scores at SoC 
2-Personal and SoC 3-Management in their concerns about READ. 

Project Management Soft ware; Timeline 

This section of the paper presents the second program 
evaluation use of tie Concerns-Based At5oDtion Model. It 
discusses the use of a new innovation, "project management 
software" in program evaluation. in recent years, several 
microcomputer software packages have become available to assist 
in program management. While this type of software was first 
developed for use in business, it provides an evaluator with a 
powerful, timesaving tool for planning, scheduling, monitoring, 
and reviewing complex projects. Project management software 
programs can be used in a variety of situations where scheduling ' 
is important, resources are precious, and tasks are contingent 
upon completic* of other tasks. The typical district program 

- 12 - 
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evaluation unit is an appropriate environment for this software. 

In July 1986, the program evaluation unit in the Portland 
Public schools began using a new project management software 
package, Timeline (c) from Breakthrough Software. The Assistant 
Director for Program Evaluation proposed that the seven members 
of the unit pilot Timeline: i) to improve the evaluation 
management process by providing more precise and timely data 
organization, and 2) to evaluate the application of microcomputer 
software to improve the efficiency and effectiveness of planning 
and monitoring program evaluations. In effect. Timeline was to 
be a unit standard for project planning and management. 

The first use of the software was to develop a comprehensive 
list of anticipated evaluation projects for 1986-87. Each 
program evaluation was set up as an individual task with a start 
and end point and a table of staff resources. In October 1986, 
as the time came to develop individual project plans, a group 
training session was conducted to acquaint unit members with the 
software and demonstrate its use. Several evaluation plans were 
completed at this work session and a November deadline and 
process for completing the remaining plans was established. As 
the individual plans were completed, they were linked together to 
form a comprehensive picture of the unit's work. Information in 
the plans included: tasks, staff resources, and schedules. 
Figure B in the appendix displays a sample Timeline Gantt chart. 

- 13 - 
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Evaluating the TmplPTn entatlon of Timeline 



-The project management software was implemented by unit 
members during 1986-87. As with any innovation, the level of 
detail and use of the process varied widely among individuals. 
One computer-generated plan consisted of over lOO tasks spread 
out over three years and involving twelve resource people; other 
plans were rough shell outlines of the evaluation process. 

The manager's goal was to measure the level of use of 
project management software at the end of the first year. The 
evaluation used two CBAM methods: i) a stages of Concern 
questionnaire given to assess individuals' feeling about th« 
Timeline program, and 2) Levels of Use interviews measured the 
behaviors of individuals regarding the project management system. 

Stage? pf Concern in January 1987, a stages of Concern 
questionnaire (SoCQ) was administered to obtain information on 
how well the program evaluation unit was doing in their efforts 
to make effective use of project management software. For this 
assessment, the SoCQ was modified^ and administered to 8 staff. - 




The SoCQ Manual strongly cautions users against modifying 
the instrument or using it with any populations other than 
teachers and administrators. Nonetheless, the program manager 
chose to make modifications since the SOCQ provided a theorv 



- 14 



16 



ERIC 



Data were collected anonymously and scored with an Apple lie 
program that produced individual graphic profiles and a composite 
profile for the group. Figure 2 displays the group SoCQ profile. 
The profile is a nonuser profile with a high Stage 0-Awareness 
concerns and moderate concerns at the other stages. Because 
Stage 2-Personal concerns are equal to stage 1-Informational 
concerns, a portion of the group may have doubts or potential 
resistance to the innovation. The Awareness score is typical of 
a user who is just becoming aware of the innovation. The 
moderate informational and personal scores indicate some interest 
in learning more about project management software, but this is 

Fig\ire 2 

Program Evaluator Concerns About Timeline 

90 I ~ 
80 




A«r taf Per lift Con Col B«f 

Stages of Concern 
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balanced with concerns about how the user will be affected by its 
use. The low level of concern for consequences is difficult to 
interpret since this is one of the categories that was modified 
from the original form. Originally, this scale pertained to 
consequences for students; for this purpose, the focus was 
changed to consequences for the work of the unit. Of course this 
change renders the norms useless. However, the individual 
variations in SoCQ profiles within the unit provide some 
indication of how the members view the software's impact on their 
work. In this interpretation, a high score indicates high 
concern for the work of the unit and the effects of Timeline on 
the work. In general, a high score on Stage 4 -Consequences is a 
positive indication of concern. Conversely, a low score would 
indicate a low level of concern for the impact of the project 
management software on the work of the program evaluation unit. 

Results of the SoC questionnaire were reviewed by the unit 
manager to plan intervention strategies and support systems 
appropriate to each individual's current concerns. 

b. L?Y^3,g of Vse The purpose of the focused LoU interviews was 
to find out the baseline use of the Timeline project management 
software in the unit at the end of the first year. The LoU 
interviews were conducted by a certified LoU interviewer with the 
seven program evaluation staff members in June, 1987. The 
standard LoU interview procedures were used; subjects were asked 
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the branching interview questions using the term "Timeline" in 
place of "the innovation." The interviews were taped and then 
rated until agreement by at least two LoU interview/ raters. 

Table 4 presents the distribution of Levels of Use of 
Timeline project management software for the Evaluation 
Department program evaluation specialists. 

Table 4 

Distribution of LoU of Timeline 
Project Management Software for Program Evaluation Unit 

(N=7) 



Level 
0 


I 


II 


III 


IVA 


IVB 


V 


VI 


N 
% 


2 

(29%) 




2 

(29%) 


1 

(14%) 


1 

(14%) 




1 

(14%) 



The results of the LoU interviews confirmed the hunches of 
the assistant director on the use of Timeline by individual unit 
members. Overall, the interviews found a lower level of use than 
the manager had anticipated; he had hoped to see more LoU IVA- 
Routine Users or at least LoU Ill-Mechanical Users, instead, the 
results indicated a wide disparity of use among members of the 
program evaluation unit. In the interviews, several individuals 
stated their concerns about the inadequate level of technical 
mastery of the software and the need for additional training. A 
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conmonly perceived weakness was that Timeline took too much time 
or was inappropriate for the type of service provided by some 
evaluators. There was agreement that the best use of project 
management software would be in larger scale resource management, 
e.g., complex longitudinal research, departmental level planning 
and budgeting, or business use. Basically, the interviews 
categorized the unit members into two groups: i) a user group 
that was attracted by what project management software offered in 
terms of planning and monitoring of evaluation activities, and 
2) an Orientation nonuser group that was acting in a coiipliance 
mode and saw the disadvantages of Timeline outweighing the 
advantages. If project management software was to be successful 
for the organization, it needed all the players using the system. 

The relationship between SoC and LoTI indicated a pattern of 
high, intense individual concerns combined with low level of use 
of project management software. The SoCQ and LoU results were 
used to develop individual and group intervention strategies 
whereby the unit manager could assist staff member in adapting to 
the software. Results help to inform Evaluation Department 
management decisions on the future use of planning and management 
software . 



As a final note, in August 1987, the unit manager mandated 
use of Timeline for all 1987-88 program evaluation projects, 
recognition of the user's concerns, support materials in the 
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form of generic evaluation planning modules were developed. This 
standard was provided to make resource comparisons across 
projects meaningful and to streamline the evaluation design 
process. Additional training was provided to the staff and a 
deadline for completion of all evaluation plans was established. 
However, when the deadline passed with only half of the program 
evaluations on a Timeline, the strong individual concerns 
overcame the innovation. The use of project management software 
continue!^* to decline. Currently, the use of Timeline is an 
option for evaluators if they and their clients find it of value 
in the planning and monitoring of program evaluations. 

Social Studies Sf-af f Developmeni^ 

The final use of the CBAM in this paper describes the 
evaluation of a K-8 social studies staff development program at 
the cluster level. During 1986-87 the Portland district adopted 
a new Scott Foresman social studies curriculum. As part of the 
adoption process, one cluster of schools provided inservice 
training to acquaint teachers with the new social studies text 
and instructional program. Four resource teachers conducted 
ongoing staff development and model teaching to individual 
teachers and grade groups in the cluster. Each resource 
teacher/trainer was assigned to work with one cf the following 
grade level groups during the year: K-1, 2-3, 4-5, or 6-8. 
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Recognizing that the Concerns-Based Adoption Model is an 
effective needs assessment process for staff development 
programs, the cluster administration chose to analyze the social 
studies inservice program as an innovation. There were two 
purposes to this effort: l) the cluster wanted to measure the 
effect of the social studies staff development, and 2) they 
wanted to assess the effect of integrating the social studies 
curriculum with the reading program. For teachers, this was a 
change in instructional focus from teaching social studies alone 
to focusing on reaoing in the social studies content area to 
encourage integration of subjects across the curricula. 



The evaluation of the social studies inservice program 
utilized the Stages of Concern questionnaire to measure teacher 
satisfaction with the training at the end of one year. In May 
1987, 87 K-8 teachers from seven schools in the cluster completed 
the SoC questionnaire with the innovation identified as "the 
social studies inservice" training program. The SoC 
questionnaires were collected anonymously and scored with an 
Apple He program that generated individual graphic profiles and 
a composite profiles by school and by grade level group. 

Figure 3 presents the teacher concerns profile in grades K-1 
and grades 2-3) for the social studies inservice. Figure 4 
presents the teacher concerns profile in grades 4-5 and grades 6- 
8 for the social studies inservice. 
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Figure 3 

Teacher Concerns About Sodal Studies Inservice by Grade 
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Figure 4 

Teacher Concerns About Social Studies Inservice by Grade 
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Overall, the profiles indicate typical »^onusers and early 
users of an innovation. The grade 4-5 group concerns indicate 
nonuse; the grade 2-3 group had moved into high SoC 3 -management 
concerns about the social studies adoption. It should be noted 
that after one year of implementation and support for che new 
social studies program, the grade level group profiles showed a 
tail up at stage 6-Refocusing which may mean the groups have 
other ideas about how to do things differently; they may also be 
negative toward the innovation. 

Results of the SoC questionnaires profiled by school group 
indicate that 57% of the cluster schools were early users of the 
innovation; 42% were nonusers. The results showed that teachers 
with high stage 1-Informational and Stage 2 -Personal concerns 
were more often clustered in schools assessed at low fidelity of 
implementation of the social studies program. 

The SoCQ composite profiles allowed cluster-level decision 
makers to assess the effects of the social studies staff 
development and plan the appropriate next level of action to 
support the curriculum adoption. 
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This paper presented applications of the Concerns-Based 
Adoption Model in program evaluation. It described the use of a 
theory-based framework in evaluating the implementation of three 
educational change efforts: an alternative language arts program 
for grades 1-8, a project management software program for 
departmental planning, and a K-8 social studies* inservice 
program. Criteria for successful implementation were identified. 
Findings varied within each program evaluation. 



The utilization of this framework was particularly valuable 
in formative evaluation. The rich descriptive data obtained in 
the interviews provided useful program improvement information. 
Often this was information the project could not have obtained on 
its own. The confidential xty of the interviews, combined with 
the two-way dialogue between teacher and evaluator, yielded high 
quality information. The interviews also verified the fidelity 
of the program implementation. This approach, used in 
conjunction with the configurations, allowed data to be gathered 
on whether or not the program was proceeding according to plan. 

Significant information for staff development was also 
provided in the teacher questionnaires. These proved useful in 
identifying areas of need and allowed decision makers to make 
data-based decisions targeted at program improvement. This 
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theory-based model can assist the evaluator in defining program 
components and interpreting teacher concerns and use. it also 
guides the program staff in developing staff development plans, 
-•nonitoring the client's use of an innovation, and designing 
practical intervention strategies to increase the effectiveness 
of the program. 



- 24 - 

26 



BIBLIOGRAPHY 



A*'.derson, S. & Balx, S. (1978). The profession and practice of 
Droaram evaluation, san Francisco, CA: Jossey-Bass Publications. 

Charters, W. W., Jr. & Jones, J. E. (1973). On the risks of 
appraising non-events in progreun evaluation. Educational 
Researcher. 2, (H) , 5-7. 

Hall, E. (1982). Viewing evaluation use as an innovation. 
Studies in Educational t^raltiat^nn ^ £(2), 185-196. 

Hall G. E. & Hord, S. M. (1987). Change In Schools! raellltatlna 
the Process, /ibany, NY: State University of New York Press. 

I.»31, G. E., tt Loucks, S. F. (1977). A developmental model for 
determining whether the treatment is actually implemented. 
American Educational Researc h Journal . 11(3), 263-276. 

Hall, G. E., Loucks, S. F. , Rutherford, W. L. , & Newlove, B. W. 
(1975). Levels of use of the innovation: A framework for 
analyzing innovation adoption. Journal o f Teacher Education ^ 
2Sl(1), 52-56. 

Hansen, J. B. & Williams, G. L. (1987 April). The use of project 

fflans^^^SmSJlfc — gQftVrarg — fac planning and monitoring program 

fiXalua&ifiUS. Paper presented at the annual meeting of the 
American Educational Research Association, Washington, D. C. 

King, J. A., & Pechman, E. M. (1984). Pinning a wave to the 
shore: Conceptualizing evaluation use in school systems. ' 
Educational Evaluation and P olicy Analvals . Fall £(3), 241-251. 



- 25 - 



27 



Loucks, S. P. & Hall, G. E. (1977). Assessing and facilitating 
the implenentation of innovations: A ni,w approach. Education;!! 
TeehnolociYr H, 18-21. 

Moilanen, c. (1988 January). 1986-87 Pv.i uatinn p^p^..^- 
ProiflCt REAP. Portland OR: Portland Public Schools. 

Pratt, H., Winters, s., & George, A. A. (1980, April). The 
gffeotg Of a goncffrns-based ininien.ftn<-;.<- iQn niar^ i^^^ 
flChigveaent of elementary aeiencA ff ^-M^t nt n Paper presented at 
the annual meeting of the American Educational Research 
Association, Boston, MA. 



Reidy, E. P., 6 Hord, S. M. (1979, April). Utilizing 

iffiPlgmgntatlffl dnta In the evaluation of « ffl ^ ithe,n«<.<n« ^y;]^-^^ 
flChigygment mpnltgrlng program . Paper presented at the annual 
meeting of the American Educational Research Association, San 
Prancisco, CA. 



Roecks, A. L., & Andrews, J. H. (1980, April). Levels of use 
IntgrVlWg; A successful format-Sv e evaluai-Mnn ^ ^"1 Paper 

presented at the annual meeting of the American Educational 
Research Association, Boston, MA. 

Rossi, P. H., Preeman, H. E., & Wright, s. R. (1979). 

Evamatipn; A systematic aonroanh. Beverly Hills, CA: Sage 

Publications. 



26 - 

28 



APPENDIX 



Project READ Configuration Checklist 
Stages of Concern Questionnaire 
Levels of Use Intein^iew Questions 
Levels of Use Rating Sheet 
Timeline (c) Sample Gantt Chart 
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Figure A 
PROJECT READ 
INSTRUCTIONAL CONFIGURATION CHECKLIST 



Project READ instructional strategies include these key elements: 
Systematic skills presentation simple to complex 

2' Use of multi-sensory learning activities 

3' Use of activity-based learning 

4. Use of direct instruction: 

Structured, sequenced instruction 

Pacing allowing for frequent practice 

Reinforcement for correct responses 

Correcting student errors 

Monitoring student progress 

Small group instruction 

Modeling generalization of skills 

5. Use of Madeline Hunter lesson plan design: 

Anticipatory set 

Statement of objectives 

Delivering information 

Modeling behavior 

Check for understanding 

Guided practice 

Independent practice 
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Project REAL 
Concerns Questionnaire 



Name (optional) 



tr^%S.i^1%,f'' ^Uits of your 



- irrelevant it««,, pjiease circ ""u" Tth' sclle oth- ir itZ,%,"^^'''^^ 
those concerns vou do have in j "^n^r items will represent 

marked higheVVtteTcale. ^ * '"'•"sity. and sh™id be 

For example: 

This statement is very true of me at this time. 0 1 2 3 4 S 6 7 

This statement is somewhat true of me no\*. 0 1 2 3 4 5 6 7 

This statement is not at all true of me at this time. 0 1 2 3 4 S 6 7 

This statement seems irrelevant to me. 0 12 3 4 5 6 7 

^ilSt y^tSlvVmenToT^tStM?:^^ 
Thank you for taking time to complete this task. 



Copyright, 1974 



Rfn rJlf* Adopting Educational Innovations/CPAM Proiect 

R5D Center for Teacher Education. The University of Texas at Austin 



- 29 - 



o 31 

ERIC 



PROJECT READ STAGES OF CONCERN 
QUESTIONNAIRE ITEMS 



Irrelevant 



Not true of me now 



5.4 s 

Somewnat true of me now 



1. 
2. 
3. 
4. 



6. 
7, 

8. 

9, 
10, 

11, 
12, 
13, 

14, 

15, 

16, 

17, 



am concerned about students' attitudes toward Project READ, 
now know of some other approaches that might work better, 
don't even know what Project READ is. 

inyself°2ach"day^^°"^ ^^""^ ^° organize 

^uld like to help other faculty in their use of Project 

have a very limited knowledge about Project READ, 
pr^f^lsiii^a? ll^T. '''''' °^ reorganization on my 

«y'L™nit1es! "^^^"^^^ 
am concerned about revising my use of Project READ. 

would like to develop working relationships with both 
our faculty and outside faculty using Project READ. 

am concerned about how Project READ affects students, 
am not concerned about Project READ. 

would like to know who will make the decisions in the 
new system. 

^uld like to discuss the possibility of using Project 

would like to know what resources are available if we 
ecide to adopt Project READ. 

READ req"^?s'' ""^ inability to r-anage all Project 

s"suJposed\S°chS^^^^^^^ °^ administration 



6 

Very true 



0 
0 
0 
0 



0 
0 



1 
1 
1 
1 

1 



2 
2 
2 
2 

2 



3 
3 
3 
3 



1 2 
1 2 



3 
3 



of me now 

4 5 6 7 

4 5 6 7 

4 5 6 7 

4 5 6 7 

4 5 6 7 

4 5 6 7 

4 5 6 7 



0 1 2 3 4 5 6 7 



0 12 3 
0 12 3 



0 12 3 
0 12 3 



0 12 3 



0 12 3 



0 12 3 



0 12 3 



4 5 6 7 
4-567 

4 5 6 7 

4 5 6 7 

4 5 6 7 

4 5 6 7 

4 5 6 7 

4 5 6 7 



01234567 



- ^ , Copyright, 1974 

Procedures for Adopting Educational Innovations/CBAM Project 
m Ceiter for Teacher Education. The University of Texas at Austin 
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Irrelevant Not true of me now Somevrf.at true^f me now ^ Very true of me now 

18. I would like to familiarize other departments or persons 
with the progress of this new approach. 

19. I am concerned about evaluating my impact on students. 

20. I would like to revise Project READ'S instructional 
approach. 

21. I am completely occupied with other things. 

22. I would like to modify our use of Project READ based 
on the experiences o£ our students. 

23. Although I don't know about Project READ, I am concerned 
about things in the area. 

J excite my students about their part in 012345-67 



25. I am concerned about time spent working with nonacademic 
problems related to Project READ. 

26. I would like to know what the use of Project READ will 
require in the immediate future. 

27. I would like to coordinate my effort with others to 
maximize Project READ'S effects. 

28. I would like to have more information on time and energy 
commitments required by Project READ. 



30. At this time, I am not interested in leamine about 
Project READ. 

31. I would like to determine how to supplement, enhance 
or replace Project READ. * 

32. I would like to use feedback from students to chance the 
program. * 



34. Coordination of tasks and people is taking too much of 
my time. 



01234567 

01234567 
01234567 

01234567 
01234567 

01234567 



this approach. 

01234567 
01234567 
01234567 
01234567 

29. I would like to know what other faculty are doing in 01234567 

01234567 



01234567 
01234567 



^ will change when lam 01234567 



using Project READ. 

01234567 

ie^'Sie ioi! '° ^^'^ ^ '^^^ 0 1 2 3 4 5 6 7 

Copyright, 1974 

Be«*?°^®^"'?* Adopting Educational Innovations/CBAM Proiect 
^ R5D Center for Teacher Education, The University of ToTas at A^tin 
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Handout # 13 



LoU Interview 



O-II/III-VI Are you currently using i 

H yes, turn page. If no, continiia. 

NO 

Have you ever used it in the past? If so, when? Why did' you stop? 



-If yes, go to PAST USERS (Below) . 

If no, continue. 

Have you made a decision to use _____ in the future? 

^^^^ If so, when will you begin use? 

Knowledge Can you describe for me as you see it? 

'^"'infojUation currently looking for any information about 
inrormation -i what kinds? For what purposes? 

Knowledge what do you see as the strengths and weaknesses of 

-_ in your situation? 

Assessing At this point in time, what kinds of quenions are 

you asking about ■> Give examples if necessary. 

^^'^'''"9 D° ever talk with others and share information about 

^ What do you share? 

Wfiat a>^e you planning witli respect to -i Can you 

tell me about any preparation or plans you have been making 
tor the use of ? 

'''ynl,t?nn^ri°" summarize for me where you see yourself right now 

(Optional) in relation to the use of ? 



PAST USERS • ^ 



Can you describe for me how you organized your use of . what oroblpmc: 

you found, v.nat its effects appeared to be on students? ^ 

SSsln^LikHiiiii^ ''^'^ d° " the 



.NOW, GO TO ABOVE SECTION. STARTING WITH QUESTION MARKED O/I-II. 
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Open-ended 



Assessing/ 
Know! edge 



Acquiring 
Information 

LoU V 



"^Sharing 
Assessing 



III/IVA/IVB 



Planning/Status 
Reporting 



r>IlI-V/VI 



YES 

Please describe for me how you use . (Ask 

sufficient questions to cover minimal criteria for use.) 

What do you see as the strengths and weaknesses of 

in your situation? (Have you made any attempt to do anything 
about weaknesses? Probe those they mentioned specifically.) 

Are you currently looking for any information about 

V/hat kind? For what purposes? 



Do you work with others in your use of 



_? Do you 



meet on a regular basis? Have you madeTny changes in your 
use of based on this coordination? 



If yes, go to LoU V Probes (Below) 

Do you ever talk with others about 
tell them? 



_? What do you 



(Have you considered any alternatives or different ways of 
doing things with tha program?) Are you doing any evaluatino. 
either formally ay informally, that would affect your use of' 
? Hove ycu received any feedback from students 

■ ? 



What 



that would affect the way you're using 

have you done with the information you got? 

Have you made any changes recently in how yov use 

What? Why? How recently? Are you considering making any 
changes? 

As you look ahead to later this year, what plans do you have 
in relation to your use of ? 

Are you considering or planning to make major modifications- 
or replace at this time? 



LoU V Probes <■ 



1. Please describe for me how you work together. (What things do you share with 
each other?) 

2. What do you see as the effects of this collaboration? 

3. Are you looking for any particular kind of information in relation to this 
collaboration? 

4. Do you talk with others about your collaboration? If so, what do you share 
with them? 

5. Have you done any formal or informal evaluation of how your collaboration is 
working? * 

6. What plans do you have for this effort in the future? 

If you have enough evidence to place the person at an LoU V, go to Question III-V/VI. 
If you do not think the person is an LoU V, go ♦•.o Question Sharing. 
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LEVEL OF USE RATING SHEET 



irrR&D/CBAM. 1976 



Tape Itt 




Site: 








Interviewer : 






Date: / 


/ 


I.D. 9: 








Rater: 






^ -— 

Level 


Knowledge 


Acquiring 
Information 


Sharing 


Assessing 


Planning 


Status 
Reporting 


Performing 


Overall LoU 


NoH'^Use 


0 


0 


0 


0 


0 


0 


0 


0 


D.P. A 


















Orientation 


I 


I 


1 


I 


I 


I 


I 


I 


D.P. B 


















Preparation 


II 


II 


II 


II 


II 


II 


II 


II 


D.P. C 


















Mechanical Use 


III 


III 


III 


III 


III 


III 


III 


III 


D.P. D-1 


















Routine 


IVA 


IVA 


IVA 


IVA 


IVA 


IVA 


IVA 


IVA 


, D.P. D-2 


















Refinement 


IVB 


IVB 


IVB 


IVB 


IVB 


IVB 


IVB 


IVB 


D.r. E 


















Integration 


V 


V 


V 


V 


V 


V 


V 


V 


D.P. F 


















i\enewcix 


VI 


VT 


VT 


VI 


VI 


VI 


VI 


VI 


User Is 
not doing: 


ND 


ND 




fin 


iin 


Nn 


Nn 




No information 
in Interview: 


NI 


NI 


NI 


NI 


m 


NI 


NI 




Past User 


Estimated past LoU 














The amount of Information In 


the Interview was: 




Insufficient 
for rating 


1 2 


3 A 5 


6 7 


very adequate 
for rating 


The Interviewee: 








does not fit 
on the chart 


1 2 


3 4 5 


6 7 


fits well 
on the chart 



ThA interviewee: 

ERIC 



was very difficult 
to interview 



2 3 4 5 6 

r 
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was no problem 
to Interview 



Figure B 

Sample Gantt Chart from Tlmellne(c) Project Management System 



TASK NAMC 



•7-ai SUMNER SCHOOL EVAL. /p 
B7-SI A PLANNING /■ 
S7-3I A.i Evaluation D«aign 
S7-S1 A. a Litaratura Raviaw 
S7-ai A.S Prapara TiMlina 
S7-S1 A. 4 Sand TiMlina to AO 
670I A. 5 Cliant Raviaw 
S7-31 C DATA COLLECTION /a 
S7-S1 c.i Cliant Oalivara Data 
•7-Sl C.a Nrita codino Schana 
07-31 C.3 Sand to Data Control 
S7-3I C.4 Oat a Loadad 
S7-3I 0 DATA ANALYSIS /a 
S7-SI O.i Mrita Fila Modula 
S7-S1 O.a Oo PfUm Fraqa 
S7-31 0.3 Raquaat Analyaaa 
S7-31 0.4 Oo Final Analyaaa 
S7-3I E REPORT MRITIN6 /a 
S7-3I E.i Nrita Oraft 
S7-3I E.a Shara Oraft: Cliant 
S7-3I E.3 Raviaa Oraft 
S7-3I e.S Raviaw Raport 
S7-3I PROJECT ENDS /a 



RESOURC 



Olw 

giw 
olw 

olw 



Olw 



trw 

Olw 
trw 

Olw 
Olw 
)bh 
MT 



LEGEND 



19B6 

JUL 



OCT NOV 



DEC 



19B7 
JAN 



FEB 



MAR 



D 



U2 



I 



1^ 



1=1 



□ 



I 



SLACK 



NILE8T0NE 
Enhancad 




Sumtp School Ualng Hoduiar Planning Duw*y 
Gary L. HUlltM 

Illuatpttaa tha um of Hlarachtcal Modular Structura 
in Tlnallna 



RUN OaTI 17NARt7 

iiBPQtTi \mm 



TIME LINE" GRAPHICS 
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